
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 1 3- 1 450 
www.usplo.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCICET NO. CONFIRMATION NO. 



09/546,213 



04/10/2000 



21171 7590 03/13/2006 

STAAS & HALSEY LLP 
SUITE 700 

1201 NEW YORK AVENUE, N.W. 
WASHINGTON, DC 20005 



Atsushi Watanabe 



392.1682/JDH 



3616 



EXAMINER 



SETH, MANAV 



ART UNIT 



PAPER NUMBER 



2625 

DATE MAILED: 03/13/2006 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 


Application No. 

09/546,213 


Applicant(s) 
WATANABE ET AL. 


Examiner 

Manav Seth 


Art Unit 

2624 





~ The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 
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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

1. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 
CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for 
continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1 .17(e) has been timely 
paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. 
Applicant's submission filed on January 11, 2006 has been entered. 

Response to Amendment 

2. Applicant's amendment under 37 C.F.R. 1.16 filed on January 11, 2006, has been entered in 
full. 

3. Applicant's arguments on pages 7-13 in amendment filed January 11, 2006, with respect to 
the rejected claim(s) 1-14 under 35 U.S.C. 103(a) respectively, have been fully considered but are not 
persuasive. See the detailed discussion in section entided in "Claim Rejections" and "Response to 
Arguments". 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 
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(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 
102 of this tide, if the differences between the subject matter sought to be patented and the prior art are such that the 
subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary 
skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

5. Claims 1, 3, 4 and 8-14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Spight 
(USPN 4,462,046) in view of Corby, Jr. et al. (USPN 5,745,387, hereafter Corby). 

Regarding claims 1 and 8, Spight discloses a teaching model generating method and device 
for image processing, in which a subject object has the same or substantially similar shape as that of 
a reference object (column 1, line 7-12), the device comprising: an image processing system with 
which current three-dimensional orientation of the subject object relative to an image pickup device 
is recognized based on a plurality of predetermined teaching models (stored data of desired object) 
of the reference object (column 3, line 26-47; column 8, line 10-15); and an image-capture system 
(video detector 40, Figure 1; column 5, line 42-48), in advance of the recognizing, generating and 
storing the plurality of teaching models (reference signals) on the basis of respective image data 
produced by taking images of said reference object from a plurality of directions, so that the image 
data respectively obtained at each of said different image pickup positions, is stored as a teaching 
model (reference signal) (column 9, line 1-19). Spight discloses "the present invention relates to use. 
of optical processing for a programmable industrial automation system wherein object 
identification, location, and orientation may be determined" (col. 1, lines 7-12) and further 
discloses robotic vision system which is capable of near real-time sensing and analysis, to identify 
and determine the location and orientation (direction) of workpieces in realistic batch 
manufacturing environment (col. 1, lines 39-44; col. 1, lines 55-60). Spight further discloses that in 
order to produce real-time identification and determination of location and orientation of parts 
(object), a viewed representation of the object is compared (pattern matching as it is done by 
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the processor) with a stored representation of the desired objects (apparently has to be similar 
shape objects) (col. 1, lines 60-65; col. 2, lines 10-35). From the above disclosure by Spight, it is clear 
that the stored image data is nothing but a plurality of teaching models based on the image data of 
the reference object taken earlier or previously, that are required to determine the identity, position 
and orientation of the object in real-time, without which there would be no real time processing 
and examiner further asserts that Spight teaches a robotic vision system with a camera (col. 1, lines 
65-68 through lines col. 2, lines 1-5) but Spight does not specifically disclose that one of the 
reference object and said image pickup device is fixed to a movable and positionable part of the 
robot or is grasped with a hand of the robot, that said robot is operated for positioning to a plurality 
of different image pickup positions and directions, or that direction information indicating the 
respective different direction is stored with the image data as a teaching model. 
However, Corby discloses an augmented reality maintenance system employing a manipulator arm 
with an archive and comparison device wherein a distal end 100 of the manipulator arm 10 is 
attached to a utility package 11, which may include a spatial imaging device such as a video camera 
(Figure 1; column 4, line 18-33). Corby also discloses a position and attitude sensing unit 21 to 
determine the position and orientation distal end 10b, which can then be used by manipulator arm 
renderer 33 to create several images of a prestored model of the manipulator arm from model 
storage 47 from several different viewpoints and environment renderer 35 to produce a number of 
images of the environment corresponding to supplied viewpoints (column 4, line 34-65). Corby 
goes on to disclose an archive and comparison device 50 that utilizes a sensor data storage device 51 
capable of storing spatial imagery with location, orientation and acquisition parameters linked to 
each image (column 6, line 29-37). Corby clearly teaches a robotic manipulator arm 10 which has a 
camera installed at the movable distal end and which is further used to render the environment 
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which further includes the inspection of machines and structures as defined in (col. 1, lines 32-40; 
col. 4, lines 22-34) and inspection of structures as well-known being a part of manufacturing process 
and here is done using the same technique of taking images at different viewpoints as described by 
Spight. Corby clearly teaches "Presently this is accomplished by moving the manipulator arm to a 
particular position (viewpoint or orientation) and videotaping the structure or device (reference 
object of similar shape) which is to be examined". At a later date the manipulator arm is positioned 
at the same site and current data (such as a video image) is compared to previous data" (col. 1, lines 
55-65; col. 6, lines 8-30- done manually; col. 6, lines 30-68 through col. 7, lines 1-25-using a system) 
and it would be apparent to one of ordinary skill in the art that if the same structure is to be 
inspected again, it has to be identified before it is examined/inspected and Corby clearly 
teaches moving the camera at a viewpoint. Corby further provides teachings of generating images at 
different orientations or directions (viewpoints) (col. 2, lines 33-40; col. 2, lines 45-52; col. 6, lines 8- 
30- done manually; col. 6, lines 30-68 through col. 7, lines 1-25-using a system), which are further 
stored and used as teaching models based on the image data (col. 4, lines 66-68 through col. 5, lines 
1-2; col.. 6, lines 10-68). Corby further clearly teaches "the sensor device 51 capable of storing 
spatial imagery with location, orientation and acquisition parameters linked to each image. 
These parameters define the identity of the site imaged (object), when it was imaged, the 
viewpoint, the modality of the imager" (col. 6, lines 35-40 with automated positioner). Further, 
Corby provides pattern matching in lines 52-67 of col. 6 through col. 7, lines 1-25 where the 
comparison or matching is done by an archive and comparison device 50, where examiner 
here asserts that pattern matching is nothing but comparing images by a computer (col. 7, 
lines 1-20, image comparison, image difference and highlighting the regions are provided by 
a system), to identify the object as defined both by Spight and Corby. Therefore, it would have 
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been obvious to one of ordinary skill in the art at the time the invention was made to fix an image 
pickup device to a movable and positionable part of a robot, operate the robot for positioning to a 
plurality of different image pickup positions and directions, and store direction information 
indicating the respective different direction with the image data as a teaching model as taught by 
Corby in order to define the identity of the site imaged, when it was imaged, the viewpoint, the 
modality of the imager and description of values relating to the image (column 6, line 37-51) and 
more emphasis added, to further enhance the object identification when object is viewed at different 
directions or viewpoints, keeping in view the 3 dimensional structure of the object, in real-time and 
further both references are directed to robotic industrial automation. 

Note also that Spight does not explicidy disclose that the recognized current orientation of 
the subject is three-dimensional (determine object's position and orientation in space ; column 8, line 
10-15) and Corby only mentions it in passing (column 2, line 6-12; column 8, line 41-45). The 
examiner takes Official Notice that determining three-dimensional orientation of an object is well 
known in the art of robotic vision systems. It would have been obvious to one of ordinary skill in 
the art at the time the invention was made to determine the three-dimensional orientation of an 
object in order to determine the orientation at which to pick up an object. 

Regarding claim 3, Spight discloses that said teaching model is a part of the image data of the 
reference object (column 9, line 10-19; column 10, line 50-63). 

Regarding claim 4, Spight discloses that said teaching model comprises data obtained by 
performing image processing on the image data of the reference object (column 9, line 10-19; 
column 10, line 50-63). 
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Regarding claims 9 and 12 (see above discussion of claim 1), Spight discloses a method of 
automatic orientation recognition, comprising: generating and storing a set of images of different 
relative orientations (arrangements; claim 12) of a subject (column 9, line 1-19), the images having 
been captured by a plurality of robotic operations corresponding to the different relative 
orientations (arrangements) of the subject (Corby; column 4, line 43-65; lines 52-67 of col. 6 through 
col. 7, lines 1-25), and associating with each image information indicating its respective relative 
orientation (arrangement) of the subject (Corby column 6, line 29-51; lines 52-67 of col. 6 through 
col. 7, lines 1-25); after the generating and storing, from a known current orientation (arrangement) 
of (an image pickup device on) a robot, capturing a current image of a workpiece that has an 
unknown orientation (arrangement) relative to an image pickup device on the robot (Corby; column 
4, line 18-33) (before the robot has come into contact with the workpiece; claim 9 only), where the 
workpiece has a shape substantially similar to the shape of the subject (Spight; column 3, line 29-38; 
column 6, line 11-13); after the capturing, using pattern matching (correlation) to match one of the 
stored images with the current image (column 7, line 28-column 8, line 15; column 9, line 59-column 
10, line 11); and after the pattern matching, (and before the robot has come into contact with the 
workpiece; claim 9 only), determining the orientation (arrangement) of the workpiece relative to the 
image pickup device on the robot based on the relative orientation (arrangement) information 
associated with the matched stored image (reference signal), and also based on the known current 
orientation (arrangement) of the robot (column 8, line 16-37; column 11, line 21-38). 

Regarding claim 10, Spight discloses automatically maneuvering the robot to the workpiece 
based at least on the determined orientation of the workpiece relative to the robot (column 8, line 
16-37). 
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Regarding claim 11, Spight discloses that the generating and storing is performed for a 
plurality of differently shaped subjects (store a plurality of configurations of each desired object; 
column 9, line 1-19), wherein the current image includes a plurality of differendy shaped workpieces, 
and wherein the pattern matching further comprises identifying workpieces from among the 
plurality of differendy shaped workpieces using the images and orientation information of the 
plurality of differendy shaped subjects (column 6, line 11-13; column 7, line 50-column 8, line 15; 
column 11, line 29-38). 

Regarding claims 13 and 14 (see above discussion of claim 1), Spight discloses a method 
comprising: robotically taking images of a subject with different three-dimensional subject-camera 
arrangements that vary in three dimensions (column 9, line 1-19; Corby, column 4, line 17-33),, and 
associating with each image or data thereof information indicating its subject-camera arrangement 
(Corby, column 6, line 29-51); then taking a current image of a workpiece shaped like the subject 
(Spight; column 3, line 29-38; column 6, line 11-13); and then before picking up the workpiece 
determining a current workpiece-camera orientation by matching (correlating) one of the images or 
data thereof with the current image (Spight, column 7, line 28-column 8, line 35; column 9, line 59- 
column 10, line 11), and using predetermined subject-camera arrangement information of the 
matched image to determine the three-dimensional orientation of the workpiece relative to the 
camera (Spight, column 8, line 16-37; column 11, line 21-38). 



6. Claims 2, 5 and 6 are rejected under 35 U.S.C. 103(a) as being unpatentable over Spight in 
view of Corby and further in view of Werth et al. (USPN 4,504,970, hereafter Werth). 
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Regarding claim 2, Spight discloses a teaching model generating device for image processing, 
in which a subject object has same or substantially similar shape as that of a reference object, the 
device comprising: an image processing system with which a current three-dimensional orientation 
of the subject object relative to an image pickup device is recogni2ed by carrying out pattern 
matching processing of an image of the subject based on a plurality of pre-determined teaching 
models of the reference object; and an image capture system, in advance of the recognizing, 
generating and storing the plurality of teaching models on the basis of respective image data 
produced by taking images of said reference object from a plurality of directions, wherein said image 
pickup device is fixed to a movable and positionable part of a robot or is grasped with a hand of the 
robot, which is operated for positioning to a plurality of different relative image pickup positions 
and directions, so that the image data respectively obtained at each of said different image pickup 
positions is stored as a teaching model (see above discussion of claim 1). 

Neither Spight nor Corby discloses that the reference object is fixed to a movable part of a 
first robot or is grasped with a hand of the first robot. Werth discloses a training controller for 
pattern processing system wherein it is suggested that an application could utilize two robot arms, 
one which holds a camera which visually guides it to observe a precise assembly point and a second 
which brings a tool or assembly within the visual field of the camera where it is visually guided 
through an operation (column 5, line 12-17). It would have been obvious to one of ordinary skill in 
the art at the time the invention was made to utilize two robot arms, one to hold a camera and one 
to hold a tool or assembly as taught by Werth in order to provide more degrees of freedom allowing 
more views of the workpiece from different directions and to provide proper alignment for mating 
parts in automated assembly operations or move a tool to a specific point on the part (column 5, line 
6-11) and more emphasis added, to further enhance the object identification when object is viewed 
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at different directions or viewpoints, keeping in view the 3 dimensional structure of the object, in 
real-time and further all the references are directed to robotic industrial automation. 

Regarding claim 5, Spight discloses that said teaching model is generated for every direction 
in which said image pickup device took the image of said reference object (column 9, line 1-19) and 
Corby discloses that said teaching model is stored in association with the information on the 
direction (column 6, line 29-41). 

Regarding claim 6, Spight discloses that said image pickup device 40 is a camera (column 5, 
line 42-48). 

7. Claim 7 is rejected under 35 U.S.C 103(a) as being unpatentable over Spight in view of 
Corby in view of Werth as applied to claim 2 above, and further in view of Ninomiya et al. (USPN 
4,611,292, hereafter Ninomiya). 

Spight, Corby nor Werth disclose that said image pickup device is a three-dimensional visual 
sensor that measures a distance between the image pickup means and a plurality of points on the 
object. Ninomiya discloses a robot vision system including a three-dimensional visual sensor whose 
image pickup means measures the distance between the image pickup means and a plurality of 
points on the object (column 4, line 28-49). It would have been obvious to one of ordinary skill in 
the art at the time the invention was made to utilize a three-dimensional visual sensor as taught by 
Ninomiya in order to determine the position and posture of an object without operation or accuracy 
being effected by contrast, color, or surface condition of the object (column 10, line 35-39). 
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Response to Arguments 

8. Applicant's arguments regarding the prior art rejections under Spight and Corby on pages 6- 
9 of the Amendment filed on April 14, 2005, have been fully considered but are not persuasive. 

9. In the 2 nd paragraph of the page 9, Applicant argues in substance: 

a. Neither Spight nor Corby, Jr et al.« teach or suggest that "one of the reference 

object and said pickup device is fixed to a movable and positionable part of a robot for 

positioning to a plurality of different image pickup position and direction ....[that ate] 
stored as a teaching model" as recited in claim 1» 8. 9 and similarly in claims 12 and 13. 

Examiner respectfully disagrees. Examiner maintains the same arguments as made in 
the previous office action mailed on 09/07/2005 on pages 9-11. 

10. In the 2 nd paragraph of the page 8, Applicant argues in substance: 

b. The newly cited portions of the Background of the Invention section in Corbv. Jr. 
et al. describe "manipulator arms ... useful for inspection of machines or structures which 
are in environments which are inaccessible or very hazardous for humans" (column 1, lines 
32-34) and the prior art inspection method that was improved upon by Corbv. Jr. et al. "to 
determine the rate of deterioration.) the manipulator arm is moved to a particular position ... 
for videotaping the structure or device which is to be examined. At a later date the 
manipulator arm is positioned at the same site and current data (such as a video image) is 
compared to previous data" (column 1. lines 55-60). All of this portion of Corbv. Jr. et al. 
teaches away from using Corbv. Jr. et al. to modify Spight or that Corbv y Jr. et al. is relevant 
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to the invention, since neither Spight nor the invention ate directed to operations in 
"environments which are inaccessible or very hazardous for humans" or M to determine the 
rate of deterioration" as in Corbv. Jr. et al. 

Examiner respectfully disagrees. Examiner does not rely on Corby to provide teachings on the 
hazardous environments for humans but to provide the teachings "that one of the reference object 
and said image pickup device is fixed to a movable and positionable part of the robot or is grasped 
with a hand of the robot, that said robot is operated for positioning to a plurality of different image 
pickup positions and directions, or that direction information indicating the respective different 
direction is stored with the image data as a teaching model". Spight clearly teaches recognizing the 
object based on a plurality of teaching models in robotic environment and examiner relies on Corby 
to show the ways the robot can be used to provide the teaching models. 

Applicant further argues in substance that "the use of the term "pattern matching" in the 
application as an operation performed by a device, not a human is consistent with definition of the 
term in the robot art". Spight clearly teaches the pattern matching using a processor rather than 
done by an operator or human (col 8, lines 1-35). Corby clearly teaches the use of archive and 
comparison (A&C) device for pattern matching (col. 6, lines 30-67 through col 7, lines 1-25, image 
comparison, image difference and highlighting the regions are provided by a system). 

All other arguments regarding all other claims are moot in view of the rejections made 

above. 
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Conclusion 

Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Manav Seth whose telephone number is (571) 272-7456. The examiner can 
normally be reached on Monday to Friday from 8:30 am to 5:00 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Bhavesh Mehta, can be reached on (571) 272-7453. The fax phone number for the organization 
where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR system, 
see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, 
contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Manav Seth 
Art Unit 2625 
March 2, 2006 
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